Procedure for mapping when capturing video streams by means of a camera

ABSTRACT

The invention relates to a procedure for mapping when capturing video streams by means of a camera, such as an IR camera, as well as to a computer program and a computer program product. According to the procedure the following items are included: a) at least two reference images are recorded in production against a black-body radiator at the same temperature for two or more separate integration times (block  3 ), b) in conjunction with the updating of an offset map, details of the integration time for which the offset map has been updated are stored (block  6 ), c) during operation, the actual integration time is compared with the integration time for the most recent updating of the offset map (block  8 ), d) the recorded reference image which lies closest to the actual integration time is selected as a reference (block  7 ), and e) a compensation per pixel for the change in the integration time is calculated by linear interpolation between the selected reference image and the most recently updated offset map, resulting in a compensation map (block  8 ). The integration time can be adapted dynamically by the procedure with access to adequate maps for correction of the offset of constituent pixels.

TECHNICAL FIELD

The present invention relates to a procedure for mapping when capturingvideo streams by means of a camera, such as an IR camera, as well as toa computer program and a computer program product.

BACKGROUND

The output signal from sensor elements in a sensor, such as a focalplane array, IR FPA, contained in a camera can vary quite considerablyas a function of the irradiated effect. Sensor elements thus require tobe calibrated with one another. The sensor elements contained in asensor in an IR camera do not behave in the same way, for example, butexhibit variations in gain and offset. In order to manage thesevariations, so-called gain maps and offset maps are recorded and storedin production. With the help of the gain map, corrections are madeduring operation for variations in gain in the individual sensorelements in a sensor. The offset map is used correspondingly to shiftthe sensor signals of the constituent sensor elements in parallel duringoperation, so that the gain curves of the detectors substantiallycoincide. To further illustrate the principles behind gain and offsetmapping, reference is made to our published US Patent Application US2011/0164139 A1.

Cameras of the IR type have traditionally made use of a plurality offixed integration times to cover the camera's dynamic range. Every suchfixed position has been provided with its own maps. The disadvantage offixed integration times is that deviations in the image quality arenecessary in order to be able to cover a particular dynamic range, sincethe image quality is optimized only at a single scene temperature.Several fixed integration time positions have been introduced in orderto increase the image quality, which improves the image quality at theexpense of higher complexity but still does not fully cover optimalintegration time positions.

An alternative to fixed integration times is to change the integrationtime dynamically. One disadvantage associated with this is that itinvolves making a shift away from the integration time which wasrelevant at the time when the maps were made. This shift away from theintegration time at the time when the maps were made gives rise to alow-frequency unevenness of the image. An extra spatial fixed-patternnoise is generated and is visible in low-contrast scenes. A previouslyknown solution to this has been to permit the integration time to bechanged only in conjunction with the offset map being updated by aso-called NUC, on uniformity correction. In order for it to functionadequately, the solution requires an NUC to be performed as soon as thescene has changed significantly, for example when panning.

For an example of the prior art where the integration time is changed,reference can be made to WO 2008/107117 A1, which describes a procedurefor changing the integration time depending on the temperature of an IRsensor.

SUMMARY OF THE INVENTION

The object of the present invention is to make available a procedurewhich adapts the integration time dynamically on the basis of the scenecontent in order to achieve maximum quality in every scene without theneed for constant updates of maps by NUC processing and without havingto store maps for a large number of fixed integration times.

The object of the invention is accomplished by a procedure that ischaracterized:

a) in that at least two reference images are recorded in productionagainst a black-body radiator at the same temperature for two or moreseparate integration times,b) in that, in conjunction with the updating of an offset map, detailsof the integration time for which the offset map has been updated arestored,c) in that, during operation, the actual integration time is comparedwith the integration time for the most recent updating of the offsetmap,d) in that the recorded reference image which lies closest to the actualintegration time is selected as a reference,e) in that a compensation per pixel for the change in the integrationtime is calculated by linear interpolation between the selectedreference image and the most recently updated offset map, resulting in acompensation map, andf) in that the compensation map is added to an incoming video image.

The use of reference images, the most recent updating of the offset mapand its integration time, as well as linear interpolation, permits acompensation map to be created which corresponds closely to the actualintegration time. The procedure permits the integration time to bechanged continuously and entirely automatically without needing toundergo a further NUC process.

The compensation map is advantageously standardized before addition bydeducting the mean value of the compensation map from all the pixels forcentring of the compensation map around zero.

Alternatively, the reference images can be standardized before theinterpolation takes place by deducting the mean value of the referenceimage from all the pixels for centring of the reference image aroundzero.

According to an advantageous procedure, specifically two referenceimages are recorded in production against a black-body radiator at thesame temperature for two separate integration times. By restricting thenumber of reference images to two, the mapping process can be keptrelatively simple from the point of view of storage and calculation.

According to the proposed procedure, the reference images can suitablybe taken for integration times within the millisecond range and with adifference in the integration time in the order of 10 milliseconds.

The invention also relates to a computer program comprising programcode, which, when the said program code is executed in a computer,causes the said computer to carry out the procedure as described abovein order to accomplish the object of the invention, as well as to acomputer program product comprising a computer-readable medium and acomputer program as described above, the said computer program beingincluded in the said computer-readable medium.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is described below in more detail by way of example withreference to the accompanying drawings, in which:

FIG. 1 depicts schematically an example of a curve which shows the scenetemperature as a function of the integration time for optimized imagequality.

FIG. 2 depicts schematically an example in the form of a block diagramof the principles for mapping according to the procedure of theinvention.

FIG. 3 depicts schematically an example of a curve which shows thesignal per pixel as a function of the integration time for a pixel whichhas undergone the mapping according to the procedure of the invention.

DETAILED DESCRIPTION OF THE EMBODIMENT

The curve 1 depicted in FIG. 1 depicts an example of the scenetemperature T as a function of the integration time t for optimizedimage quality. Two fixed integration times t₁ and t₂ are plotted in theFigure. According to the known method, an individual map is connected toeach integration time. In the event that another suitable integrationtime is used, a shift will take place from the integration times forwhich the maps were produced, and low-frequency unevennesses of theimage will occur as a result. According to the proposed procedure formapping, which is described in more detail with reference to FIG. 2,compensation maps are generated which offer relevant mapping along thescene temperature curve between fixed references.

The principles for mapping according to the procedure of the inventionare now described with reference to FIG. 2.

An incoming video stream is available through a block 2. The stream canbe supplied from the sensor part of an IR camera, the said sensor partcomprising, for example, one or a plurality of focal plane arrays. Theactual integration time for the video stream is designated with t_(p).

A block 3 marks the availability of two reference images taken inproduction. The images have been taken on a black-body radiator at thesame temperature for two different integration times. The firstreference image is designated here as B_(ref1) and was taken with anintegration time t_(ref1), which in one example here can amount to 3 ms.The other reference image is designated as B_(ref2) and was taken withan integration time t_(ref2), which in the example here can amount to 14ms. The reference image B_(ref1) is marked as being stored in block 4,while reference image B_(ref2) is marked as being stored in block 5.

During the operation of an IR camera, calibrations of the camera areperformed during a known process known as NUC, non uniformitycorrection. A block 6 indicates this NUC process. In this NUC process,the integration time, t_(nuc), with which the NUC process was carriedout is saved in addition to the most recent map. The block 6 thusprovides both the most recent map and the most recent integration timet_(nuc).

A block 7 manages the choice of the next reference image. If

|t _(ref1) −t _(p) |<|t _(ref2) −t _(p)|,

reference image B_(ref1) is selected because it lies closest to the newactual integration time t_(p). Otherwise, reference image B_(ref2) isselected because it is then considered to lie closest to the new actualintegration time t_(p).

If the integration time is to be changed, the actual integration timet_(p) is compared with the integration time t_(nuc) from the most recentNUC, and the reference which lies closest to the new actual integrationtime t_(p) is selected as a reference image and is provided by the block7.

In the block 8, a compensation per pixel is calculated by linearinterpolation between the selected reference image, either referenceimage B_(ref1) or B_(ref2), and the map from the most recent NUC. Theresult of the calculation is a map that is referred to in thisdescription as a compensation map. A block 9 undertakes standardizationof the correction map before it is added via an adder 10 to the incomingvideo stream from the block 2 in order to deliver a video stream througha block 11. The output video stream can be connected, for example, to adisplay for presentation or to a memory medium for storage. A displayand a memory medium are not shown in the Figure.

Alternatively, standardization of the reference image can be undertakenbefore interpolation, and two possible positions of a block for thestandardization of reference image B_(ref1) are indicated with dashedblocks 13 and 15 in FIG. 2. Two dashed blocks 14 and 15 likewiseindicate two possible positions of a block for the standardization ofreference image B_(ref2).

The result of linear interpolation per pixel is illustratedschematically in FIG. 3. The signal S per pixel is shown on the curve 12as a function of the integration time t. The integration times t_(ref1)and t_(ref2) for the reference images B_(ref1) and B_(ref2) are plottedalong the time axis t together with the integration time t_(nuc) for themost recent NUC. The actual integration time t_(p) is plotted inaddition. In the illustrated example, reference image B_(ref1) is closerto the actual integration time t_(p) than reference image B_(ret2). Inline with what has been described with reference to the block 8 in FIG.2, a linear interpolation is made between the reference image B_(ref1)with an integration time t_(ref1) and the map from the most recent NUCwith an integration time t_(nuc).

The invention is described above with reference to a schematic blockstructure depicted in FIG. 2. This description must be regarded only asan explanation of the fundamental structure. The execution of thedescribed functions can be performed in many ways within the frameworkof the invention, and attention is brought particularly to the use of acomputer program for the implementation of the functions.

The invention is not restricted to the procedures described above asexamples, but may be subjected to modifications within the scope of thefollowing patent claims.

1. A method for mapping when capturing video streams with a camera themethod comprising, recording at least two reference images in productionagainst a black-body radiator at a same temperature for two or moreseparate integration times; storing, in conjunction with an updating ofan offset map, details of an integration time for which the offset maphas been updated; comparing, during operation, an actual integrationtime with the integration time for the most recent updating of theoffset map; selecting a recorded reference image, which lies closest tothe actual integration time, as a reference; calculating a compensationper pixel, for the change in the integration time, by linearinterpolation between the selected reference image and the most recentlyupdated offset map, resulting in a compensation map; and adding thecompensation map to an incoming video image.
 2. The method according toclaim 1, wherein the compensation map is standardized before addition bydeducting a mean value of the compensation map from all the pixels forcentring of the compensation map around zero.
 3. The method according toclaim 1, wherein the reference images are standardized before theinterpolation takes place by deducting the mean value of the referenceimages from all the pixels for centring of the reference image aroundzero.
 4. The method according to claim 3, wherein two reference imagesare recorded in production against a black-body radiator at the sametemperature for two different integration times.
 5. The method accordingto claim 4, wherein the reference images are taken for integration timeswithin the millisecond range and with a difference in the integrationtime in the order of 10 milliseconds.
 6. A computer program comprisingprogram code, wherein when the program code is executed in a computerthe computer is configured to carry out the method of claim
 1. 7. Acomputer program product comprising a computer-readable medium and thecomputer program according to claim 6, wherein the computer program isincluded in the computer-readable medium.
 8. The method of claim 1,wherein the camera is an infrared camera.
 9. An infrared cameraconfigured to perform the method of claim
 1. 10. A method of processinginfrared data, the method comprising: comparing, during operation, anactual integration time with an integration time for the most recentupdating of an offset map; selecting a recorded reference image, whichlies closest to the actual integration time, as a reference; calculatinga compensation per pixel, for the change in the integration time, bylinear interpolation between the selected reference image and the mostrecently updated offset map, resulting in a compensation map; and addingthe compensation map to an incoming video image.
 11. The methodaccording to claim 10, wherein the compensation map is standardizedbefore addition by deducting a mean value of the compensation map fromall the pixels for centring of the compensation map around zero.
 12. Themethod according to claim 10, wherein the reference images arestandardized before the interpolation takes place by deducting the meanvalue of the reference images from all the pixels for centring of thereference image around zero.
 13. The method according to claim 12,wherein two reference images are recorded in production against ablack-body radiator at the same temperature for two differentintegration times.
 14. The method according to claim 13, wherein thereference images are taken for integration times within the millisecondrange and with a difference in the integration time in the order of 10milliseconds.
 15. A non-transitory computer-readable medium encoded withexecutable instructions for performing a method of mapping whencapturing video streams with a camera, wherein the method comprises:comparing, during operation, an actual integration time with anintegration time for the most recent updating of an offset map;selecting a recorded reference image, which lies closest to the actualintegration time, as a reference; calculating a compensation per pixel,for the change in the integration time, by linear interpolation betweenthe selected reference image and the most recently updated offset map,resulting in a compensation map; and adding the compensation map to anincoming video image.